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1.   Introduction 

The  Ultracomputer  project  has  been  studying  various  aspects  of  an 
ideal  highly  parallel  computer,  the  "paracomputer" ,  proposed  by 
Schwartz  [UC] .  Gottlieb  [UCN12,  UCN13,  UCN21]  wrote  a  paracomputer 
simulator,  WASHCLOTH,  which  implemented  the  essential  synchronization 
primatives  of  a  paracomputer.  Denelcor  has  since  developed  the  HEP 
computer,  a  highly  parallel  computer,  which  uses  a  simpler,  and 
possibly  less  efficient  synchronization  technique  [HEP].  In  order  to 
explore  the  actual  effect  of  choice  of  synchronization  techniques,  we 
have  modified  WASHCLOTH  to  include  the  HEP  synchronization  primatives. 
All  the  features  of  the  May  1982  version  of  WASHCLOTH  have  been 
retained. 

The  new  features  are  based  on  designation  of  a  region  of 
"asynchronous"  memory,  for  which  flags  are  maintained  which  indicate 
which  words  are  "full"  (last  stored  into)  or  "empty"  (last  read  from). 
Attempts  to  read  from  an  empty  asynchronous  location  cause  the  pe 
involved  to  hang  until  the  location  becomes  full.  Attempts  to  write 
into  a  full  asynchronous  location  cause  the  pe  involved  to  hang  until 
the  location  becomes  empty.  Routines  are  provided  to  test  for  full  or 
empty  and  force  variables  to  a  known  state. 


[*]Permanent   Address:    Brookhaven   National  Laboratory,   Chemistry 
Department,  Upton,  L.I.,  New  York  11973. 
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To  simplify  programming,  and  to  be  compatible  with  HEP  FORTEAN,  pe 
scheduling  routines  have  been  provided.  Execution  always  starts  with 
only  one  pe  active  and  ends  when  none  are  active.  Pes  are  assigned  to 
subroutines  calls  held  in  a  queue  on  a  first  come,  first  served  basis. 

As  a  side  effect  of  the  above  changes,  some  existing  WASHCLOTH 
features  have  been  extended.  Indivisible  mode  now  has  access  to  all  of 
memory,  without  restriction,  since  entry  to  indivisible  mode  causes  a 
copy  from  private  memory  to  real  (i.e.  unsimulated)  memory.  Also,  an 
entry  has  been  provided  to  revert  to  full  machine  speed  execution  when 
in  indivisible  mode,  so  that  I/O  can  be  done  at  reasonable  cost. 

This  new  simulator  is  called  RINSE,  simply  because  that  seemed  a 
right  thing  to  have  follow  WASHCLOTH. 

Preliminary  tests  of  empty-full  synchronization  with  RINSE  show 
the  penalty  for  the  use  of  this  synchronization  rising  approximately 
where  the  use  of  any  synchronization  at  all  becomes  a  dominant  factor. 
More  extensive  tests  of  this  behavior  are  planned. 
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2.   Definition  of  the  RINSE  virtual  machine 

RINSE  simulates  up  to  32  CYBER-like  processors  with  some  added 
instructions  and  special  subroutine  calls.  The  added  instructions  are 
those  of  WASHCLOTH.  The  subroutine  calls  follow  CDC  FORTRAN  extended 
calling  conventions.  Each  processor  (pe)  has  its  own  register  set, 
including  its  own  program  address  register,  and  sees  its  own  private 
copy  of  low  memory. 

Each  pe  may  be  either  active  or  idle.  On  entry,  only  one  pe  is 
active.  It  is  the  responsiblity  of  the  user  code  to  activate  the 
additional  pes  it  requires.  Up  to  128  such  activations  may  be  queued, 
pending  a  free  pe .  An  active  pe  may  become  hung  because  it  is  trying 
to  activate  another  pe  and  no  pe  or  queue  entry  is  available,  or 
because  it  is  trying  to  access  an  "asynchronous"  variable  which  is  in 
the  wrong  state.  If  all  active  pes  become  hung,  no  progress  can  be 
made  and  RINSE  terminates. 

Any  pe  may  lock  out  execution  by  all  other  pes  by  entering 
"indivisible"  mode.  Until  it  returns  to  divisible  mode,  no 
synchronization  with  other  pes  is  necessary  or  possible.  All  memory 
below  a  specified  boundary  address  is  private  to  each  pe,  i.e.,  each  pe 
sees  its  own  copy  of  that  memory.  To  avoid  conflicts  with  the  CYBER 
operating  system,  the  RINSE  virtual  machine  considers  it  an  error  to 
attempt  to  store  into  private  memory  below  a  trap  address  located  in 
the  simulator  code,  except  in  indivisible  mode. 

All  memory  above  the  private  memory  boundary  is  shared  among  all 
pes,  i.e.  it  is  public.  Within  the  public  area,  variables  below  a 
designated  boundary  have  no  special  synchronization  control,  but  those 
above  that  boundary,  which  are  called  "asynchronous"  variables,  can 
only  be  written  into  when  "empty",  and  can  only  be  read  from  when 
"full".  As  noted  above,  a  pe  which  attempts  to  use  an  asynchronous 
variable  in  conflict  with  these  rules  simply  hangs  until  the  condition 
is  satisfied . 
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The  full  CYBER  central  instruction   set   is   simulated,   with  the 
addition  of  the  following  operations  and  calls: 

Operations 


Read  pe  number 

Indivisible 

Divisible 

No  private 

Private 

Replace-Add 


00  1  0  k 
00  2  0  k 
00  3  0  k 


(RPN  Bk) 
(INDIV  k) 
(DIV  k) 


00  4  0  202624B  (NOPVT) 

00  5  0  0    (PVT) 

00  i  j  k   (REPAD  Xl,Bj+k) 


Calls 


EMPTY(arg)        Logical  function,  returns  .true,  if  arg 

is  empty  and  asynchronous 
FULL(arg)  Logical  function,  returns  .true,  if  arg 

is  full  or  not  asynchronous 
PURGE ( argl , . . . , argn) 

Subroutine,  forces  arguments  to  empty  if  they 

are  asynchronous 
FILL(argl, . . . ,argn) 

Subroutine,  forces  arguments  to  full  if  they 

are  asynchronous 


RINSE  [UCN47] 


page-5 


CREATE ( subr , argl , 


IDLEPE 
RESUME 


IRPN( dummy) 
INDIV(k) 

NOSIM 

SIM 

DlV(k) 

NOPVT 

PVT 

IREPAD(i, jexpr) 

REPAD(x,yexpr) 


. .,argn) 

Subroutine,  causes  call  to  subr(argl , . . . ,argn) 
to  be  queued  for  execution  by  some  pe  when 
available 

Subroutine,  causes  calling  pe  to  go  idle 
Subroutine,  called  from  a  routine,  say  A,  which 
was  itself  called  by  another  routine,  say  B,  to 
cause  the  routine  B  to  resume  execution  before 
A  executes  a  RETURN  (requires  a  free  pe) 
Function,  returns  ordinal  of  executing  pe 
Subroutine,  forces  indivisible  mode,  charge  k 
per  cycle  simulated 

Subroutine,  forces  indivisible  mode  and  reverts 
to  full  machine  speed  execution 
Subroutine,  returns  to  simulated  execution  in 
indivisible  mode 

Subroutine,  returns  to  divisible  mode  with 
charge  k  added  to  cycle  count 
Subroutine,  turns  off  private  memory  handling 
Subroutine,  restores  private  memory  handling 
Function,  performs  a  replace-add  of  integer 
expression  jexpr  on  integer  variable  i. 
Function,  performs  a  replace-add  of  real 
expression  yexpr  on  real  variable  x 


The  operations  and  the  routines  IRPN,  INDIV,  DIV,  NOPVT,  PVT,  IREPAD 
and  REPAD  have  essentially  the  same  meaning  as  with  WASHCLOTH,  except 
that  in  the  RINSE  virtual  machine,  entry  into  indivisible  mode  is 
sufficient  to  allow  access  to  all  of  memory  for  system  calls,  so  NOPVT 
and  PVT  are  not  needed.  They  are  retained  for  compatability  with 
existing  code.  The  new  routines  EMPTY,  FULL,  PURGE,  FILL,  CREATE, 
IDLEPE,  and  RESUME  have  the  effect  of  one-cycle  instructions  embedded 
within  the  virtual  machine.  However,  they  do  not  have  any  equivalent 
operation  code. 
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3.   Calling  RINSE 

In   order   to   call  RINSE,  the  user  must  establish  a  COMMON  block 
with  the  following  structure: 

COMMON  AVAIL ( n)  ,PUBDAT(ni)  ,ASYNDT(k) 

and  make  the  following  type  declarations 

EXTERNAL  subr 
LOGICAL  RINSE, rtest 

where  n  is  bigger  than  the  number  of  pes  to  be  simulated  plus  one, 
times  the  number  of  locations  in  private  memory  that  may  change  while 
in  divisible  simulated  mode,  m  is  the  size  of  non-asynchronous  public 
memory  desired,  and  k  is  the  size  of  asynchronous  public  memory 
desired. 

The  execution  of  RINSE  is  started  by 

rtest=RINSE(npe, AVAIL, PUBDAT, subr, lowtrap,hightrap,ASYNDT) 

where  npe  is  the  number  of  pes  to  simulate,  subr  is  the  subroutine  in 
which  the  simulation  is  to  start,  lowtrap  is  some  variable  in  public 
memory  marking  the  lowest  point  to  trap  stores  and  hightrap  is  the 
highest  point.  On  return  from  RINSE,  rtest  will  be  true  if  RINSE 
executed  without  error  (e.g.  it  did  not  deadlock  on  full/empty). 
Unlike  WASHCLOTH,  RINSE  returns  even  if  it  did  have  an  error. 
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After  RINSE   execution,  the  COMMON  QRINSE  will  have  the  following 
information 

icycnt     number  of  machine  cycles  of  simulated  execution 
avail      address  of  next  free  location  in  AVAIL 
botpub     address  of  lowest  public  location 
lowtrp     address  of  lowest  address  to  be  trapped 
hitrp      address  of  highest  address  to  be  trapped 
botsyn     address  of  lowest  asynchronous  location 
ilc(32)    program  address  registers  of  pes  (0-31) 
ipos(32)    instruction  parcel  pointers  of  pes  (0-31) 
atable(8,32) 

A-registers  of  pes 
xtable(8,32) 

X-registers  of  pes 
btable(8,32) 

B-registers  of  pes 
iact       active  pe  flag  word,  one  bit  per  pe  (0  on  exit) 
ipecy(32)   active  instruction  cycle  count  for  each  pe 
lcnt(32)   internal  scratch  for  detecting  deadlocks 
ihng(32)    count  of  cycles  spent  hung  and  active  for  each  pe 
icdly(32)   count  of  words  copied  in  creating  memory  image 

for  a  pe  when  it  starts  up 
ipeasn(32)  count  of  number  of  times  each  pe  was  assigned  a 

task 
bavail     base  address  of  AVAIL 
isind      flag,  non-zero  in  indivisible  mode 

ipeq(131)   queue  of  CREATE  requests  (count,  in,  out,  128  requests) 
pecl(96)    dummy  calls  used  when  dequeuing  CREATE  entries 
ibtab(2185) 

bit  table  for  private  memory  and  asynchronous  memory 
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The  user  must  provide  a  routine, 

TRAP(pen,iclk.,loc,oldval,newval) 

which  will  be  called  for  stores  within  the  range  between  the  memory 
trap  addresses,  accepting  the  arguments  pen  (the  number  of  the  pe 
executing  the  store),  iclk  (the  number  of  machine  cycles  executed),  loc 
(the  address  into  which  the  store  is  being  done),  oldval  (the  original 
value),  and  newval  (the  new  value). 

When  used  with  FTN,  no  other  special  dummy  routines  are  required. 
However,  when  used  with  M77,  the  user  must  provide  the  routines  PDUMP 
and  REGDMP,  since  M77  does  not  provide  a  PDUMP  and  the  REGDMP  in  NYULIB 
requires  the  FTN  library.  In  most  cases,  dummy  routines  will  suffice, 
since  PDUMP  and  REGDMP  are  called  only  on  RINSE  errors.  Since  M77 
provides  a  post-mortem  dump,  making  PDUMP  abort  can  be  used  to  get 
error  information.  With  M77,  the  user  must  also  avoid  the  name  OUTPUT 
as  a  subroutine  name  in  calls  to  CREATE,  due  to  a  bug  in  the  compiler. 

As  an  example  of  the  way  to  call  RINSE,  consider  the  following 
approach  to  adding  20  numbers  drawn  from  [HEP]: 
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PROGRAM  RINSET( INPUT, OUTPUT) 
COMMON/QRINSE/NCY , MAVAIL , MBP , MLT , MHT 

*  ,MBS,ILC(32),KPOS(32),MAT(256),MXT(256),MBT(256) 

*  , lACT, IPECY(32) ,LCNT(32 ) , IHANG(32) , ICDLY(32) , IASN(32 ) , IBA, ISD 

*  ,IPEQ(131),IPECL(96),IBTAB(2185) 

COMMON  AVAIL(2048),BOTPUB,PDATA(100),LOWT,HIT, 

*  BOTSYN,A(2G),S(5),SUM 
EXTERNAL  TEST 

MPE=10 

CALL  RINSE ( MPE , AVAIL , BOTPUB , TEST , LOWT , HIT , BOTS YN ) 

PRINT  90,NCY 
90   FORMAT (1X,*CYCLES  *IIO) 

PRINT  100, (IHANG(I ),!=!, MPE) 
100      F0RMAT(1X,*HANGS  *  lOIlO) 

PRINT  110,(IPECY(I),I=1,MPE) 
110      FORMAT (IX,* CYCLES  *  lOIlO) 

CALL  EXIT 

END 

SUBROUTINE  TEST 

EXTERNAL  INPUT, OUTPUT, ADD 

COMMON  AVAIL ( 2 04 8), BOTPUB, PDATA( 100), LOWT, HIT, 

*  BOTSYN,A(20),S(5),SUM 
CALL  NOSIM 

PRINT  90 
90       FORMAT(*  THIS  IS  A  TEST  *) 

CALL  DIV(O) 

DO  100  1=1,20 
100      CALL  PURGE(A(I)) 

DO  110  1=1,5 
110      CALL  PURGE(S(I)) 

CALL  PURGE (SUM) 

CALL  CREATE(INPUT,A,20) 

DO  30  1=1,5 
30       CALL  CREATE(ADD,A(4*I-3),S(I),4) 

CALL  CREATE(ADD,S,SUM,5) 

CALL  CREATE (OUTPUT, SUM, 1) 
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RETURN 

END 

SUBROUTINE  ADD(V ,ANS .NUMBR) 

DIMENSION  V(l) 

TEMP=0.0 

DO  20  J=l, NUMBR 

TEMP=TEMP-'-V(J) 
20       CONTINUE 

ANS=TEMP 

RETURN 

END 

SUBROUTINE  INPUT (A, N) 

DIMENSION  A(N) 

CALL  NOSIM 

PRINT  lll.N 
111    FORMAT (IX,* CALL  TO  INPUT  N=*I5) 

CALL  DIV(O) 

CALL  NOSIM 

READ  100, A 

DO  120  1=1, N 
120    CALL  FILL(A(I)) 
100      FORMAT(FIO.O) 

CALL  DIV(O) 

RETURN 

END 

SUBROUTINE  OUTPUT(A,N) 

DIMENSION  A(N) 

DUMMY=A(1) 

CALL  NOSIM 

PRINT  100, A 
100      FORMAT(1X,F20.10) 

CALL  DIV(O) 

RETURN 

END 
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SUBROUTINE  TRAP(IPEN, ICLK.LOC ,OLDV, NEWV) 

RETURN 

END 

This  code  works  because  each  pe  using  a   partial   sum   in   asynchronous 
memory  hangs  until  that  partial  sum  becomes  available. 

4.   Implementation  Information 

The  May  1982  version  of  WASHCLOTH  was  used  as  the  basis  for  RINSE. 
The  handling  of  memory  access  was  changed  from  use  of  special  data  in 
memory  to  use  of  a  bit  table  ( ibtab) ,  which  has  a  1  for  any  word  which 
is  private  and  heterogeneous  among  pes,  or  which  is  public, 
asynchronous,  and  full.  When  a  word  is  stored  in  private  memory  in 
divisible  mode,  the  bit  for  that  word  in  ibtab  is  set,  and  a  block  of 
memory  from  avail  is  used  to  holds  the  data.  The  block  has  one  word 
for  each  pe  plus  one  word  for  the  address  represented.  l-Then 
indivisible  mode  is  entered,  the  data  for  the  executing  pe  is  copied 
from  avail  to  the  appropriate  locations.  On  return  from  indivisible 
mode,  the  data  is  copied  back  to  avail.  This  removes  the  need  for 
noprivate  mode  and  avoids  all  conflict  with  system  I/O  routines  and 
loaders,  at  the  expense  of  some  time  lost  on  entry  and  exit. 

Extension  for  the  new  RINSE  entries  was  done  by  trapping  return 
jumps  to  designated  locations,  rather  than  by  adding  operation  codes. 
When  the  rj  instruction  is  simulated,  the  target  address  is  compared 
with  a  table  (RJTL)  in  RINSE,  and  if  a  match  is  found,  execution 
continues  within  RINSE  with  a  charge  of  one  instruction  cycle.  This 
gives  these  calls  the  flavor  of  monitor  calls,  rather  than  applications 
calls. 

In  order  to  save  time  in  test  runs,  the  entries  NOSIM  and  SIM  were 
added  which  allow  the  simulation  to  be  turned  off  completely  for  I/O 
and  then  back  on  again.  Naturally,  while  the  simulation  is  turned  off, 
none  of  the  WASHCLOTH  operation  codes  are  available.  However,  the 
calls   other  than  IRPN,  INDIV,  NOPVT,  and  PVT  are  available,  since  they 


RINSE  [UCN47]  page-12 

all  check  a  flag  in  RINSE  to  see  if  they  are  running   in   simulated   or 
real  mode. 

On  detected  deadlocks  and  other  errors,  PDUMP,  rather  than  DUMP  as 
in  WASHCLOTH,  is  called  to  dump  QRINSE,  and  as  much  of  avail  as  was 
used.   Control  then  returns  to  the  calling  program. 

To  match  the  flavor  of  HEP,  execution  starts  with  only  one  pe 
active.  Users  requiring  more  need  only  enter  indivisible  mode,  call 
CREATE  as  many  times  as  wished  and  then  return  the  divisible  mode.  All 
the  created  pe  calls  will  start  at  the  same  time  if  sufficient  idle  pes 
are  available. 

For  example,  to  simulate  the  effect  of  WASHCLOTH  of  May  1982,  the 
following  code  can  be  used: 

SUBROUTINE  DRY 

C 

C     THIS  IS  THE  INTERFACE  ROUTINE  WHICH  ALLOWS  A 

C     WASHCLOTH  STYLE  STARTUP  UNDER  RINSE 

C 

COMMON  /WASHCOM/  NPE,IAVL, IBTPB,ITMN, ITMX.ISL 

DIMENSION  IBE(l) 

IRA=-L0CF(IBE)+1 

CALL  NOSIM 

DO  100  1=1, NPE 
100   CALL  CREATE (IBE( I SL+IRA)) 

CALL  SIM 

CALL  DIV(O) 

CALL  IDLEPE 

END 

SUBROUTINE  WASHCLO (NUMPE , AVL , BOTPUB , lASTRT , MINTRP , MAXTRP ) 
C 

C     THIS  IS  A  WASHCLOTH  TO  CALL  WHEN  USING  RINSE 
C     IT  SETS  UP  THE  COMMONS  EXPECTED  BY  [UNC23]  SYNCHRONIZATION 
C     ROUTINES,  AND  CALLS  RINSE  TO  START  AT  ROUTINE  DRY 
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C     DRY  STARTS  ALL  PES  AT  lASTRT,  WHICH  MUST  BE  THE 

C     LOCATION  OF  A  WORD  AFTER  A  SUBROUTINE  ENTRY  POINT,  SINCE 

C     THE  LOCATION  BEFORE  WILL  GET  A  RETURN 

C 

COMMON  /WASHCOM/  NPE , lAVL, IBTPB , ITMN, ITMX, ISL 

EXTERNAL  DRY 

DIMENSION  IBE(l) 

IRA=-L0CF(IBE)+1 

NPE=NUMPE 

IAVL=LOCF(AVL) 

IBTPB=LOCF(BOTPUB) 

ITMN=LOCF(MINTRP) 

ITMX=LOCF(MAXTRP) 

ISL=IASTRT-1 

CALL  RINSE(NUMPE,AVL,BOTPUB,DRY,MINTRP,MAXTRP,IBE(IRA+377777B)) 

RETURN 

END 

With  this  code,  the  IDOLOOP  and  SYNC  routines  for  converting 
scientific  codes  to  parallel  execution  can  be  run  [UCN23].  As  we  note 
below,  there  is  a  slight  skew  in  starting  one  of  the  pes,  since  we  are 
short  one  at  the  time  of  return  to  divisible  mode. 
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4.   Preliminary  Test  Results. 

While  we  will  report  on  more  extensive  tests  in  future  notes,  it 
seems  appropriate  to  include  a  information  on  preliminary  tests  of 
RINSE  here.  A  linear  equation  solver,  converted  by  Korn  for  WASHCLOTH 
was  tested  under  WASHCLOTH,  under  RINSE  with  replace-add 
synchronization,  and  under  RINSE  with  empty-full  synchronization.  The 
code,  DRY,  above  was  used  to  hook  RINSE  to  the  test  program  for  the 
replace-add  test.  The  only  detectable  differences  between  WASHCLOTH 
and  RINSE  with  replace-add  synchronization  were  in  actual  cpu  time, 
where  RINSE  was  a  factor  of  2  to  3  slower,  in  the  number  of  virtual 
machine  cycles,  where  13  cycles  were  lost  in  the  DRY  code  and  65  cycles 
were  lost  due  to  one  pe  getting  a  late  start  in  leaving  DRY  (the  one 
executing  DRY  had  to  go  idle  after  the  others  were  already  running) , 
and  in  memory  requirements,  where  RINSE  required  about  3K  decimal 
additional  words. 

The  conversion  to  empty-full  synchronization  was  done  by  changing 
replace-adds  in  the  conversion  routines  into  fetch  -  add  one  -  store 
sequences  on  asynchronous  variables.  This  has  the  same  effect  as  a 
replace-add,  but  extends  over  a  few  machine  cycles  during  which  other 
pes  may  hang.  The  code  was  tried  on  16  equations  in  16  unknowns  for  16 
right  hand  sides,  for  1,  2,  4,  8  and  16  pes.  The  total  execution 
cycles  were  as  follows: 
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//pe  empty-full  replace7add  cycles  lost 

1  418148  417677  47l' 

2  216404  215600  804 
4  115678  114947  731 
8  65468  64577  891 

16       40770       39618   1162 

In  this  test  case,  363  of  the  cycles  lost  on  the  empty-full 
synchronization  were  in  initial  setup  of  the  asynchronous  storage 
locations.  The  rest  were  true  conflicts,  with  the  knee  in  the  curve 
falling  somewhere  between  8  and  16  pes,  which  is  also  the  approximate 
location  of  the  knee  of  the  over-all  efficiency  curve.  One  might 
expect  this  behavior,  since  we  can  expect  to  be  penalized  for  using 
empty-full  synchronization  when  multiple  pes  are  all  waiting  at  the 
same  synchronization  point  for  significant  times,  which  is  also  a  case 
in  which  over-all  efficiency  falls. 
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